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Cyclic  Use  of  Prototyping 


•  "Fly-Before-Buy" 


•  Packard  Commission 
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Weapons  System  Acquisition  Reform  Act  of  2009 

and  Prototyping 


•  Competitive  prototyping  of  systems  or  critical 
subsystems  before  Milestone  B  approval 

•  If  competitive  prototyping  is  waived  by  MDA,  a 
prototype  must  still  be  produced  before  MS  B 
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Defense  Acquisition  Management  System  201 3 

Preliminary  Design  Review  (PDR) 
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Mandatory  for  MDAPs 


PDR  Before  Milestone  B 

PDR  After  Milestone  B 

•  Planned  for  in  Acquisition  Strategy 
•PDR  Report  provided  to  MDA  at  MS  B 
•Includes  recommended  requirements  trades 

resulting  from  prototyping  and  critical 
technology  demonstrations 

•  Mandatory  for  MDAPs  and  DASD(SE) 
participates 

•Planned  for  in  Acquisition  Strategy 
•PDR  Report  provided  to  MDA  prior  to  Post  PDR 
Assessment 

•Reflects  requirements  trades 
•At  Post  PDR  Assessment,  MDA  considers  PDR 
report;  determines  action(s)  required  to  achieve 
APB  objectives  and  issues  ADM 
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Section  2366b  of  Title  10,  United  States  Code,  requires  certification  that  the  technology  in  the  program  has 


been  “demonstrated  in  a  relevant  environment”  prior  to  IVSIestone  B.  (This  is  interpreted  as  TRL  6.) 


Joint  Lightweight  Tactical  Vehicle  (JLTV) 


USA/USMC 
Contract  Type 
TD  Contract  Costs 
Requirements 
TMRR  Phase 
Prelim  Design  Rev 
TRL  (at  MS  B) 


BAE  Systems  AM  General/GDLS  Lockheed  Martin 

Various  Various  CPFF 

$62.9  M  $61.3  M  $53.4  M 

CDD,  15  March  2012 
27  months 
June  -  July  2009 

5  (underbody)  /  TD  protoypes  built  on  assembly  line 


Littoral  Combat  Ship  (LCS) 


USN  General  Dynamics  Lockheed  Martin 


Contract  Type 
TD  Contract  Costs 
Requirements 
TMRR  Phase 

Preliminary  Design  Review 
TRL  (at  MS  B) 


FPI  FPI 

$575  M  $537  M 

validated  CDD,  June  2008;  10  KPPs 
72  months 

July  2003  (prior  to  MS  A) 

?  (integration  w/mission  packages)  /  9  (seaframe) 


Small  Diameter  Bomb  (SDB)  II 


USAF  /  USN 
Contract  Type 
TD  Contract  Costs 
Requirements 
TMRR  Phase 
Critical  Design  Rev 
TRL  (at  MS  B) 


Boeing  /  Lockheed 

CPFF 
$161.4  M 


Raytheon 

CPFF 
$161.4  M 


validated  CDD,  June  2009;  5  KPPs 
42  months 

within  6  months  of  MS  B  (June  2010) 
6  (Program  Office  Estimates) 


Research  Issue 


Determine  if  DoD  Instruction  5000.02  policies  for  Major 
Defense  Acquisition  Programs  (MDAPs)  relating  to 
competitive  prototyping,  technology  readiness,  and 
Preliminary  Design  Review  (PDR)  prior  to  Milestone  (MS)  B 
are  having  the  desired  effect  on  program  outcomes. 

Research  questions: 

1.  Does  the  knowledge  from  competitive  prototyping  and 
a  PDR  conducted  prior  to  MS  B  result  in  better  decisions 
relative  to  requirements,  design,  and  resources? 

2.  What  are  the  effects  of  the  competitive  prototyping, 

technology  readiness,  and  PDR  policies  on  program  costs 
and  program  schedules?  9 


Research  Methodology 

•  Cost  growth  was  determined  by  comparing  the  original  Program 
Acquisition  Unit  Cost  (PAUC)  with  the  current  PAUC  estimate, 
calculated  to  the  same  base-year  dollars,  as  reported  in  the  Unit 
Cost  Report  (UCR)  of  the  annual  Selected  Acquisition  Reports 
(SARs)  for  2011  and  2012 

•  Annual  SARs  also  identify  if  programs  have  suffered  an  Acquisition 
Progrorn  Baseline  (APB}  threshold^  schedule  breach 

•  Government  Accountability  Office  (GAO)  survey  data  was  used  to 
identify  programs  that  have  demonstrated  technology  maturity  on 
prototypes  in  a  relevant  environment  (Technology  Readiness  Level 
6)  and  have  conducted  a  preliminary  design  review  prior  to 
Milestone  B 
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Life  Cycle  Cost  Composition 


Research  Methodology 

Descriptive  statistics  are  used  to  analyze  cost  growth  (percent 
change  to  date  in  PAUC)  and  schedule  breaches  for  the  MDAPs  that 
have  conducted  competitive  prototyping  and  PDR  activities. 

Similar  descriptive  statistics  are  used  to  analyze  the  balance  of  the 
MDAPs  included  in  a  particular  annual  SAR  submission. 

The  percentage  of  programs  that  have  negative  cost  growth 
(negative  percent  change  to  date  in  PAUC)  from  each  population  is 
compared.  The  population  with  the  highest  number  of  negative 
cost  growth  programs  is  preferred. 

The  percentage  of  programs  that  suffered  an  APB  schedule  thres¬ 
hold  breach  from  each  population  is  compared.  The  population 
with  the  lowest  percent  of  schedule  breaches  is  preferred.  n 


Research  Results 


PAU£  Cost  Growth  Results.  Based  upon  data  from 
the  2011  and  2012  SAR,  programs  that 
demonstrated  technology  maturity  on  prototypes 
in  a  relevant  environment  (TRL  6)  and  conducted 
a  preliminary  design  review  prior  to  Milestone  B 
were  more  often  to  show  negative  PAUC  cost 
growth. 

This  result  was  seen  in  all  DoD  Components. 
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Research  Results 


Table  2.  Programs  Costing  Less,  Selected  Acquisition  Report,  December  31 , 201 1 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Army 

6 

7 

86 

3 

12 

25 

Navy 

7 

15 

47 

6 

20 

30 

Air  Force 

5 

10 

50 

4 

15 

27 

Def  Agency 

1 

1 

100 

2 

9 

22 

Total 

19 

33 

57 

15 

56 

27 

Table  3.  Programs  Costing  Less,  Selected  Acquisition  Report,  December  31, 2012 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Army 

5 

8 

62 

4 

12 

33 

Navy 

8 

18 

44 

4 

20 

20 

Air  Force 

3 

8 

38 

6 

17 

35 

Def  Agency 

0 

0 

0 

4 

5 

80 

Total 

16 

34 

47 

18 

54 

33 
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Research  Results 


Schedule  Threshojcj  Breach  Results.  Based  upon 
data  from  the  2011  and  2012  SAR,  programs  that 
demonstrated  technology  maturity  on  prototypes 
in  a  relevant  environment  (TRL  6)  and  conducted 
a  preliminary  design  review  prior  to  Milestone  B 
did  not  suffer  fewer  APB  schedule  threshold 
breaches. 

This  result  was  seen  in  all  DoD  Components 
except  the  Air  Force. 
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Research  Results 


Table  4.  Program  Schedule  Breach,  Selected  Acquisition  Report,  December  31 , 201 1 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Army 

2 

7 

28 

2 

12 

17 

Navy 

4 

15 

27 

5 

20 

25 

Air  Force 

4 

10 

40 

6 

15 

40 

Def  Agency 

1 

1 

100 

4 

9 

44 

Total 

11 

33 

33 

17 

56 

30 

Table  5.  Program  Schedule  Breach,  Selected  Acquisition  Report,  December  31, 2012 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Army 

3 

8 

38 

4 

12 

33 

Navy 

6 

18 

33 

3 

15 

Air  Force 

2 

8 

25 

7 

17 

41 

Def  Agency 

0 

0 

0 

0 

5 

0 

Total 

11 

34 

30 

14 

54 

26 
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Future  Research 


•  To  remove  some  of  the  uncertainty  in  the  cost  growth  metric,  compare  PAUC 
based  upon  the  original  cost  estimate  with  actual  PAUC.  Actual  PAUC  can  be 
determined  from  contracts  found  in  the  Defense  Cost  and  Resource  Center 
(DCARC)  database. 

•  To  remove  some  of  the  uncertainty  in  the  schedule  slippage  metric,  compare  the 
original  schedule  estimate  with  actual  schedule  performance  data.  Actual 
schedule  performance  data  for  this  comparison  should  be  available  in  the  DCARC 
database  or  Defense  Acquisition  Management  Information  Retrieval  (DAMIR). 

•  Finally,  the  challenge  in  using  cost  growth  and  schedule  slippage  metrics  is  to  tie 
them  back  to  the  use  of  competitive  prototyping  (to  reveal  technology  readiness) 
and  the  use  of  an  early  PDR.  The  knowledge  from  these  activities  and  how  that 
knowledge  is  applied  will  tell  us  whether  these  policies  have  had  an  effect.  To 
that  end,  more  detailed  surveys,  such  as  those  conducted  annually  on  selected 
weapon  systems  by  the  GAO,  will  aid  in  helping  establish  the  cause-effect 
relationship  between  policy  and  program  outcomes. 
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